<html>
<head>
<title>OSPFv2 Testsuite - Documentation</title>
<link rel="stylesheet" href="testsuite.css" type="text/css">
</head>
<body>
<br><hr>
<div align="center"> 
<a href="http://ssfnet.d-jacob.net">Home</a> | 
<a href="index.html">Up</a> | 
<a href="index.html">Previous</a> | 
<a href="ospfv2.html">Next</a>
</div>
<hr><br>
<h1>Conceptual framework for testing</h1>

As mentioned before, it is crucial for a network simulator like SSFNet, 
that all implemented protocols comply to their respective specifications. 
Therefore, all protocols must be tested extensively. Since the most parts of 
SSFNet are regularly improved and completed, there is also a strong need for 
regression testing. This avoids that protocols which worked with one SSFNet 
version do not work any longer with a new release. Because of this, all 
protocols provided with SSFNet have a set of tests, which verify the 
behavior of these protocols.<p>

The ISO 9646 standard provides basic concepts for conformance testing of protocols.
It defines standardized test architectures and procedures. A protocol implementation
usually provides services at a particular layer and uses services of the layer
below. The protocol is called an <i>N-service provider</i>. The lower layer is the
<i>N-1 service</i>. All protocol implementations must be tested as black boxes. Tests
are driven by the observation of input/output behavior and not by the observation of
internal states. A component, which stimulates and observes an implementation at the
level of the N-1 service is called a <i>lower tester</i> (LT). A component, which uses
the N-service provided by the implementation itself to test the implementation, is called
an <i>upper tester</i> (UT).

These concepts can be easily transferred into conformance testing within the SSFNet 
framework. Since the OSI methods are independent of any particular system, the test 
procedure needs not to be modified. The following abstract test architecture can be used
to test arbitrary protocols in SSFnet:<p>

<div align="center"><img src="pictures/test_arch.jpg" width=300 height=119><br>&nbsp;<br>
<b>Figure 1</b> - SSFNet testing architecture</div><p>

In SSFNet, the protocol implementation (IUT) is realized as a protocol session. 
The upper tester (UT) can also be realized as a protocol session in the protocol 
graph of he IUT (e.g. the UT for a TCP session could be a HTTP session). The lower 
testers (LT) are placed in their own protocol graphs and can also be implemented 
in appropriate protocol sessions. The data exchanged between the testers and the 
IUT are protocol messages. The underlying service can be any lower level protocol 
that has already been implemented and tested in SSNet (e.g. IP, UDP or TCP). A closer 
look at the OSI test procedure shows, that there are four main phases:<p>

<ol>
<li> Specification of test cases.
<li> Implementation of an executable test suite.
<li> Execution of the test suite.
<li> Evaluation of test results and assignment of a verdict.
</ol>

The first phase is completely independent of SSFNet, since it only depends on the protocol 
standard. The result is an abstract test suite, which must be implemented for the SSFNet 
environment. The upper and lower testers are used to bring the IUT from one state to 
another. This has to be done to verify, if an implementation correctly reacts on valid 
input. But also the correct reaction on invalid or erroneous input must be validated. 
So the testers must model the valid behaviors as well as the invalid ones. To model 
standard (valid) behavior, it is sufficient for many test scenarios to provide an appropriate 
DML scenario. But more advanced tests, which verify the behavior of an IUT under various 
error conditions, cannot always be modeled only by the means of DML. This strongly depends 
on the configuration possibilities of the IUT. Test behaviors, which cannot be modeled in 
DML, must be implemented in specialized testers. So the executable test suite will be a 
collection of DML scenarios and SSF models (e.g. implemented in Java) which model appropriate 
test behaviors.<p>

In the OSI testing methodology, each test case has a preamble, which brings the IUT from 
an initial state into an appropriate state to run the test (e.g. establish a connection). 
After that, the test is run. Finally, there is a postamble, which brings the IUT back 
into the initial state. In the SSFNet framework, this postamble can be omitted, since each 
test case is executed in a different simulator run. That guarantees, that the IUT is 
definitely in the initial state, when a test begins.<p>

The execution of the test suite will produce a set of test logs. Since the implementation 
is tested as a black box, the only source of information for the evaluation of the test 
results are the packets exchanged between the testers and the IUT, which can be observed 
at the PCOs. Therefore the testers must log these packets in an appropriate way (e.g. 
dump packets in {\em tcpdump} format). However, for some tests (e.g. verification of 
routing tables in routing protocols) it could be useful to have logs of internal 
datastructures as well. For this, the simulation log can be used as a source of 
information as well. Since the test suites should be independent of a specific implementation 
in order to reuse them for other implementations of the same protocol, this possibility should 
be sparely used. <p>

After the execution of the test suite, the logs must be analyzed and a verdict must be 
assigned. This can either be done manually or automatically. When a user installs SSFNet 
on a machine, he might want to validate if SSFNet works properly in this environment. 
Since a user does not necessarily have knowledge on how to analyze the logs, the automatic 
evaluation of logs should be preferred.<p>
<br><hr>
<div align="center"> 
<a href="http://ssfnet.d-jacob.net">Home</a> | 
<a href="index.html">Up</a> | 
<a href="index.html">Previous</a> | 
<a href="ospfv2.html">Next</a>
</div>
<hr><br>
<i>Last update: 2002-02-05, Dirk Jacob (<a href="mailto:dirk@d-jacob.net">dirk@d-jacob.net</a>)</i>
</body>
</html>